Search for: All records

Visualizing optimization landscapes has resulted in many fundamental insights in numeric optimization, specifically regarding novel improvements to optimization techniques. However, visualizations of the objective that reinforcement learning optimizes (the "reward surface") have only ever been generated for a small number of narrow contexts. This work presents reward surfaces and related visualizations of 27 of the most widely used reinforcement learning environments in Gym for the first time. We also explore reward surfaces in the policy gradient direction and show for the first time that many popular reinforcement learning environments have frequent "cliffs" (sudden large drops in expected reward). We demonstrate that A2C often "dives off" these cliffs into low reward regions of the parameter space while PPO avoids them, confirming a popular intuition for PPO’s improved performance over previous methods. We additionally introduce a highly extensible library that allows researchers to easily generate these visualizations in the future. Our findings provide new intuition to explain the successes and failures of modern RL methods, and our visualizations concretely characterize several failure modes of reinforcement learning agents in novel ways. 

Abstract BackgroundAtlantic tarpon (Megalops atlanticus) are a highly migratory species ranging along continental and insular coastlines of the Atlantic Ocean. Due to their importance to regional recreational and sport fisheries, research has been focused on large-scale movement patterns of reproductively active adults in areas where they are of high economic value. As a consequence, geographically restricted focus on adults has left significant gaps in our understanding of tarpon biology and their movements, especially for juveniles in remote locations where they are common. Our study focused on small-scale patterns of movement and habitat use of juvenile tarpon using acoustic telemetry in a small bay in St. Thomas, US Virgin Islands. ResultsFour juvenile tarpon (80–95 cm FL) were tracked from September 2015 to February 2018, while an additional eight juveniles (61–94 cm FL) left the study area within 2 days after tagging and were not included in analysis. Four tarpon had > 78% residency and average activity space of 0.76 km²(range 0.08–1.17 km²) within Brewers Bay (1.8 km²). Their vertical distribution was < 18 m depth with occasional movements to deeper water. Activity was greater during day compared to night, with peaks during crepuscular periods. During the day tarpon used different parts of the bay with consistent overlap around the St. Thomas airport runway and at night tarpon typically remained in a small shallow lagoon. However, when temperatures in the lagoon exceeded 30 °C, tarpon moved to cooler, deeper waters outside the lagoon. ConclusionOur results, although limited to only four individuals, provide new baseline data on the movement ecology of juvenile Atlantic tarpon. We showed that juvenile tarpon had high residency within a small bay and relatively stable non-overlapping daytime home ranges, except when seasonally abundant food sources were present. Fine-scale acoustic tracking showed the effects of environmental conditions (i.e., elevated seawater temperature) on tarpon movement and habitat use. These observations highlight the need for more extensive studies of juvenile tarpon across a broader range of their distribution, and compare the similarities and differences in behavior among various size classes of individuals from small juveniles to reproductively mature adults. 

Abstract: We resolve three problems concerning the three entire-leaved species of Neotropical Trema. 1) Phylogenetic and morphological results showed that T. integerrima (Beurl.) Standl., T. domingensis Urb., and T. laxiflora Lundell are a single species; therefore, we combine them under a single name. 2) The date of publication gives priority to T. integerrima, as the basionym Sponia integerrima Beurl. was published in 1856. However, the type specimen, Billberg 308 (S), is not a species of Trema but Pouzolzia obliqua (Wedd.) Wedd. (Urticaceae); therefore, Sponia integerrima Beurl. is a new synonym of P. obliqua. The next available name for the entire-leaved species is T. domingensis, published in 1912. 3) The type of T. domingensis, Fuertes 312, is a mixed collection. Specimens of Fuertes 312 held at B, NY, U, and USD are T. domingensis, while those at P, G, US, and Z are Celtis trinervia Lam. We provide a historical review to understand how these problems arose and persisted unnoticed for decades.